{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Novice/Deep%20Learning%20Design%20Patterns%20-%20Workshop%20-%20Chapter%20III%20-%202.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Design Patterns - Code Labs\n",
    "\n",
    "## Lab Exercise #3 - Get Familiar with Hyperparameters\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Chapter III: Training Foundation\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Hand setting Epochs and Mini-Batches\n",
    "    2. Use ImageDataGenerator for batch generation.\n",
    "    3. Finding good learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Epochs and Mini-Batches\n",
    "\n",
    "In this section, we will hand-roll our own code (vs. builtin feeders) to feed the training data for training. We will need to handle the following:\n",
    "\n",
    "    1. Set a mini-batch size (128) and calculate how many batches will be in the training data.\n",
    "    1. Set the number of epochs (number of times we pass the full training data for training)\n",
    "    2. Randomly shuffle the training data on each epoch.\n",
    "    3. Iterate through the training data on batch at a time.\n",
    "    \n",
    "You fill in the blanks (replace the ??), make sure it passes the Python interpreter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10\n",
    "import random\n",
    "\n",
    "# Let's use the CIFAR-10 dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# We will use a mini-batch size of 128\n",
    "batch_size = 128\n",
    "\n",
    "# Calculate the total number of mini-batches in an epoch\n",
    "# HINT: It has something to do with the (mini) batch size\n",
    "batches = len(x_train) // ??\n",
    "\n",
    "# Let's use a seed so we can randomly shuffle both the pixel data and labels the same shuffle.\n",
    "seed = 101\n",
    "\n",
    "# Let's do 5 passes (epochs) over the dataset\n",
    "epochs = 5\n",
    "for epoch in range(epochs):\n",
    "    # Shuffle the dataset at the beginning of each epoch\n",
    "    # HINT: We have to shuffle the image data and labels from the training data\n",
    "    random.seed(seed)\n",
    "    random.shuffle(??)\n",
    "    random.seed(seed)\n",
    "    random.shuffle(??)\n",
    "    # Set a new seed for the next shuffle\n",
    "    seed += random.randint(0, 100)\n",
    "    \n",
    "    # Iterate (sequential) through the shuffled training data, one batch at a time.\n",
    "    for batch in range(batches):\n",
    "        # Get the next batch of data\n",
    "        # HINT: if the begin of the batch is at location X, then the end is X + batc\n",
    "        x_batch = x_train[batch * batch_size:(batch+??) * batch_size]\n",
    "        y_batch = y_train[batch * batch_size:(batch+??) * batch_size]\n",
    "        print(\"Epoch\", epoch+1, \"Batch\", batch+1)\n",
    "        \n",
    "print(\"Done - the last line above this should be: Epoch 5, Batch 390\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ImageDataGenerator and Batch Generation\n",
    "\n",
    "In this section, we will use the **Keras** ImageDataGenerator to automatically generate out mini-batches (vs. hand generating them), and shuffling the training data on each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "# Let's use the CIFAR-10 dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# We will use a mini-batch size of 128\n",
    "batch_size = 128\n",
    "\n",
    "# Calculate the total number of mini-batches in an epoch\n",
    "batches = len(x_train) // batch_size\n",
    "\n",
    "# instantiate an Image Data generator object\n",
    "# HINT: Image Data generator is a big giveaway\n",
    "datagen = ??()\n",
    "\n",
    "# Let's do 5 passes (epochs) over the dataset\n",
    "epochs = 5\n",
    "for epoch in range(epochs):\n",
    "    \n",
    "    # Use generator to create batches\n",
    "    # HINT: The method is about flowing data from in-memory (vs. on-disk)\n",
    "    batch = 0\n",
    "    for x_batch, y_batch in datagen.??(x_train, y_train, batch_size=batch_size, shuffle=True):\n",
    "        batch += 1 # Keep track of the number of batches so far.\n",
    "        print(\"Epoch\", epoch+1, \"Batch\", batch)\n",
    "        \n",
    "        # At the end of the training data, let's loop around for the next epoch.\n",
    "        if batch == batches:\n",
    "            break\n",
    "            \n",
    "print(\"Done - the last line above this should be: Epoch 5, Batch 390\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate\n",
    "\n",
    "Let's show how to do short epochs to get a feel on what might be the right learning rate for your training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential, optimizers\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "import numpy as np\n",
    "\n",
    "# Let's use the CIFAR-10 dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# Normalize the pixel data\n",
    "x_train = (x_train / 255.0).astype(np.float32)\n",
    "x_test  = (x_test  / 255.0).astype(np.float32)\n",
    "\n",
    "# One-hot encode the labels\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "\n",
    "\n",
    "def convNet(input_shape, nclasses):\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                     activation='relu',\n",
    "                     input_shape=input_shape))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dense(nclasses, activation='softmax'))\n",
    "    return model\n",
    "\n",
    "# Create a simple CNN and set learning rate very high (0.1))\n",
    "# HINT: how would you abbreviate learning rate?\n",
    "model = convNet((32, 32, 3), 10)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=optimizers.Adam(??=0.1),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Let's take a fraction of the training data to test the learning rate (2%)\n",
    "x_tmp = x_train[0:1000]\n",
    "y_tmp = y_train[0:1000]\n",
    "\n",
    "# Let's run 3 epochs at learning rate = 0.1\n",
    "model.fit(x_tmp, y_tmp, epochs=3, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Argh, it's horrible. The loss on the first epoch is high (14.0+) and then never goes down - like it's stuck.\n",
    "\n",
    "Hum, okay now you experiment with different learning rates to find one where the loss goes down rapidly and a steady increase in accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = convNet((32, 32, 3), 10)\n",
    "# Pick your own learning rate until the results are good.\n",
    "# HINT: It's going to be a lot smaller than 0.1\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=optimizers.Adam(lr=??),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Let's run 3 epochs at your learning rate\n",
    "model.fit(x_tmp, y_tmp, epochs=3, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Lab Exercise"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
